[test](be) Migrate variant index storage debugpoint tests#64348
Draft
eldenmoon wants to merge 8 commits into
Draft
[test](be) Migrate variant index storage debugpoint tests#64348eldenmoon wants to merge 8 commits into
eldenmoon wants to merge 8 commits into
Conversation
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: This adds a focused BE unit-test fixture for storage index lifecycle coverage around inverted index probe events, patched tablet schemas, variant sparse/nested metadata, and COUNT_ON_INDEX no-need-read-data behavior. The tests cover index add/drop and build/drop on text and variant path indexes, missing required variant columns, V1/V2 index storage files, rowset map updates after schema patching, and red/green validation that COUNT_ON_INDEX reads only rows filtered by an applied inverted index. The implementation records index probe events for applied, fallback, and not-attempted column predicates, preserves existing SegmentIterator debug points, skips missing physical columns only when nullable/default columns make that valid, and resolves field_pattern variant index build/drop against materialized paths.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `PATH=/tmp/codex-clang-format16:$PATH build-support/clang-format.sh`
- `PATH=/tmp/codex-clang-format16:$PATH build-support/check-format.sh`
- `git diff --check && git diff --cached --check`
- `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*'`
- Attempted `env CLANG_TIDY_BINARY=/mnt/disk1/claude-max/ldb_toolchain20/bin/clang-tidy build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN`, but local clang-tidy could not analyze Doris headers due toolchain/PCH diagnostics (`stddef.h` not found from `slice.h` and existing `core/types.h` NOLINT parsing). A clang-tidy-16 probe also failed with incompatible PCH AST.
- Behavior changed: No
- Does this need documentation: No
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29590 ms |
Contributor
TPC-DS: Total hot run time: 169390 ms |
### What problem does this PR solve? Issue Number: close apache#64205 Related PR: apache#64205 Problem Summary: Add a focused BE unit test that reproduces the variant sparse stats limit reader bug. The test writes a variant column with one materialized path, an exact sparse path v.b, and a hidden child path v.b.c that is omitted from sparse path statistics after the configured limit is reached. Reading v.b through an explicit path-column schema currently returns NULL for the row whose value only exists under b.c, exposing the missing hierarchical read behavior without applying the production fix. ### Release note None ### Check List (For Author) - Test: Unit Test - Ran Target system: Linux; Target arch: x86_64 Python 3.11.13 Check JAVA_HOME version Apache Maven 3.9.12 (848fbb4bf2d427b72bdb2471c22fced7ebd9a7a1) Maven home: /usr/local Java version: 17.0.16, vendor: Oracle Corporation, runtime: /mnt/disk6/common/jdk-17.0.16 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.14.0-694.el9.x86_64", arch: "amd64", family: "unix" cmake version 3.31.8 CMake suite maintained and supported by Kitware (kitware.com/cmake). ccache version 4.5.1 Get params: PARALLEL -- 39 CLEAN -- 0 ENABLE_PCH -- ON EXTRA_BE_MODULES -- Build Backend UT install datasketches-cpp to thirdparty path before build backend ut Update datasketches-cpp submodule ... -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/contrib/datasketches-cpp/build/Release Install the project... -- Install configuration: "Release" -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/version.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/binomial_bounds.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bounds_binomial_proportions.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ceiling_power_of_2.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/common_defs.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/conditional_back_inserter.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/conditional_forward.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/count_zeros.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/inv_pow2_table.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kolmogorov_smirnov_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kolmogorov_smirnov.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/memory_operations.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/MurmurHash3.h -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/optional.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sorted_view_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sorted_view.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/serde.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/xxhash64.h -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/hll.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/AuxHashMap.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CompositeInterpolationXTable.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/hll.private.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketchImplFactory.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponHashSet.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponList.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CubicInterpolation.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HarmonicNumbers.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll4Array.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll6Array.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll8Array.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllArray.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketchImpl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllUtil.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/coupon_iterator.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/RelativeErrorTables.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/AuxHashMap-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CompositeInterpolationXTable-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponHashSet-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponList-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CubicInterpolation-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HarmonicNumbers-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll4Array-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll6Array-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll8Array-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllArray-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketch-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketchImpl-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllUnion-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/coupon_iterator-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/RelativeErrorTables-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/compression_data.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_common.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_compressor.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_compressor_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_confidence.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_util.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/icon_estimator.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kxp_byte_lookup.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/u32_table.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/u32_table_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_helper.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_helper_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/frequent_items_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/frequent_items_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/reverse_purge_hash_map.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/reverse_purge_hash_map_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_a_not_b.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_a_not_b_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_jaccard_similarity.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_comparators.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_constants.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_helpers.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_update_sketch_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_update_sketch_base_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union_base_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection_base_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_set_difference_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_set_difference_base_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_jaccard_similarity_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bounds_on_ratios_in_sampled_sets.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bounds_on_ratios_in_theta_sketched_sets.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/compact_theta_sketch_parser.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/compact_theta_sketch_parser_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bit_packing.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sample.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sample_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_intersection.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_intersection_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_a_not_b.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_a_not_b_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_jaccard_similarity.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_of_doubles_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_intersection.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_intersection_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_a_not_b.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_a_not_b_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_common.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_compactor.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_compactor_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/count_min.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/count_min_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/density_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/density_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tdigest.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tdigest_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bloom_filter.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bloom_filter_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bloom_filter_builder_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bit_array_ops.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/lib/DataSketches/cmake/DataSketches.cmake -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/lib/DataSketches/cmake/DataSketchesConfigVersion.cmake -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/lib/DataSketches/cmake/DataSketchesConfig.cmake Update apache-orc submodule ... Update clucene submodule ... -- Make program: /usr/bin/make -- Use ccache: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache and -DCMAKE_C_COMPILER_LAUNCHER=ccache -- Extra cxx flags: -- GLIBC_COMPATIBILITY is ON -- USE_LIBCPP is OFF -- USE_JEMALLOC is OFF -- USE_UNWIND is ON -- ENABLE_PCH is ON -- USE_AVX2 is ON -- Build type is ASAN_UT -- Build target arch is x86_64 -- DORIS_HOME is /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework -- THIRDPARTY_DIR is /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed -- make test: ON -- make benchmark: OFF -- build fs benchmark tool: OFF -- build task executor simulator: OFF -- build file cache lru tool: OFF -- build gensrc if necessary make: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc' make -C script make[1]: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/script' /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/script/gen_build_version.sh make[1]: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/script' make -C proto make[1]: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/proto' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/proto' make -C thrift make[1]: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift' /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/bin/thrift -I /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift -I /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift/../build//thrift/ --gen cpp:moveable_types,no_skeleton -out /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift/../build//gen_cpp --allow-64bit-consts -strict /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift/Normalization.thrift make[1]: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift' make: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc' -- Configuring incomplete, errors occurred!; expected failure before the apache#64205 fix: v.b values are scalar-0, scalar-1, scalar-2, scalar-3, NULL and do not include child-0. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Migrate variant array inverted index writer debug point coverage from the variant_p0 regression suite into the BE index storage test framework. The new test drives typed variant array rowset writes, verifies array writer field counts through a scoped debug point, covers null elements, empty arrays, null arrays, top-level null Variant rows, mixed typed JSON elements, and checks index file materialization across cumulative compaction. The shared index storage test utility now supports scoped debug points, array typed variant path templates, and direct Variant-column batches with shape validation for future typed Variant storage tests.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*'
- ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*' (26/27 passed; IndexStorageLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated failed and reproduced in single-test rerun with unchanged failure)
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Split Variant index storage lifecycle coverage out of the non-Variant index storage lifecycle test file so text/non-Variant and Variant storage cases can evolve independently. Add a storage-level migration for the variant_p0 multi-index compaction debug-point scenario by verifying that equality predicates on a predefined Variant string path choose the plain field-pattern index, not the parent, phrase, sibling, or array path indexes, before and after cumulative compaction.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh`
- `./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.VariantPathEqualityChoosesStringIndexAfterCompaction'`
- `./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
- `./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*'`
- `./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` (expected failure, reproduces hidden child loss before fix)
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Split pure index storage schema patch unit tests out of the lifecycle test files. The lifecycle files now focus on rowset write/read/index/compaction behavior, while text and variant schema patch construction checks live in dedicated files. This keeps future migrations from variant debug point and storage tests easier to place without mixing schema-only checks with lifecycle coverage.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh`
- `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageSchemaPatchTest.*:IndexStorageVariantSchemaPatchTest.*'`
- `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
- Red test, expected failure retained: `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64205 Problem Summary: Add storage-level coverage for typed Variant field-pattern inverted indexes across cumulative and full compaction. The test writes typed glob predefined paths, verifies that an INT predicate uses only the matching field-pattern index before compaction, then reloads the compacted rowset and verifies the same index behavior after compaction. Shared Variant index assertion helpers are moved into the index storage UT utility layer so later Variant index storage cases can reuse them instead of duplicating probe-event scans. ### Release note None ### Check List (For Author) - Test: Unit Test - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantFieldPatternIndexTest.*'` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` - Expected red test: `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` fails with `has_hidden_child` false. - Behavior changed: No - Does this need documentation: No
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29853 ms |
Contributor
TPC-DS: Total hot run time: 169463 ms |
### What problem does this PR solve? Issue Number: None Related PR: apache#64205 Problem Summary: Extend the index storage unit-test fixture to pass common expression pushdown contexts into rowset readers, then add a separate Variant expression-pushdown test for IS NULL and IS NOT NULL over a field-pattern indexed typed path. This keeps expression-level index coverage separate from field-pattern column-predicate and compaction coverage, and lets future Variant expression tests reuse the same write/read/assert framework. ### Release note None ### Check List (For Author) - Test: Unit Test - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh` - `git diff --cached --check` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantExprPushdownTest.*'` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64205 Problem Summary: Add storage-level unit coverage for Variant array_contains common expression pushdown over a field-pattern array inverted index. The test migrates the array_func.array_contains debugpoint coverage from variant_p0 into the index storage fixture by asserting the index bitmap cardinality, rows filtered by the inverted index, and returned rows without relying on SQL profile polling. ### Release note None ### Check List (For Author) - Test: Unit Test - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh` - `git diff --check` - `git diff --cached --check` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantExprPushdownTest.*'` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` - Behavior changed: No - Does this need documentation: No
eldenmoon
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #64205
Problem Summary: Build a reusable BE index-storage unit-test fixture and migrate the Variant
variant_p0debugpoint/index/compaction coverage that belongs at storage-reader/writer level into focused unit tests. The migrated coverage includes array inverted-index writer debugpoint checks, Variant path index lifecycle and compaction coverage, field-pattern typed index coverage for cumulative/full compaction, common expression pushdown forIS NULL/IS NOT NULL, andarray_containscommon-expression index filtering. Variant and non-Variant coverage is split into independent test files, with shared rowset write/read/reload/compact/index-probe helpers kept inindex_storage_test_util.The
bloom_filter_must_filter_dataprofile/debugpoint portion invariant_p0/predefine/test_types_with_indexes_profile.groovyremains regression/bloom-filter scope rather than inverted-index storage fixture scope. Existing BE tests cover bloom-filter reader/writer primitives separately; this PR migrates the inverted-index storage paths. The Cloud schema-change transaction-conflict debugpoint remains regression-scope because it validates cloud schema-change job conflict handling rather than BE rowset/index storage behavior. The sparse-stats #64205 reproducer is kept as an intentional red BE UT and is excluded from the green storage/index verification command until the product fix lands.Release note
None
Check List (For Author)
PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.shgit diff --checkgit diff --cached --checkenv DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantExprPushdownTest.*'(2 tests passed)env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'(36 tests passed)